Indications of incommensurate spin fluctuations in doped triangular antiferromagnets 
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The incommensurate spin fluctuation of the doped triangular antiferromagnet is studied within the 
t-J model. It is shown that the commensurate peak near the half-filling is split into six incommen- 
surate peaks in the underdoped and optimally doped regimes. The incommensurability increases 
with the hole concentration at lower dopings, and saturates at higher dopings. Although the incom- 
mensurability is almost energy independent, the weight of these incommensurate peaks decreases 
with energy and temperature. 
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The high-T c cuprate superconductors exhibit many un- 
usual properties 1,2 , one of the most striking being the 
anomalous incommensurate antiferromagnetism in the 
normal state in the underdoped regime 3,4 . These unusual 
properties are closely related to the fact that cuprate su- 
perconductors are doped Mott insulators, obtained by 
chemically adding charge carriers to a strongly correlated 
antifcrromagnetic (AF) insulating state 1,2 . Neutron- 
scattering measurements show that when the commensu- 
rate AF long-range-order (AFLRO) phase is suppressed 
and the hole concentration exceeds 3%, incommensurate 
dynamical short-range magnetic correlations appear 3,4 , 
with four peaks located at the reciprocal space positions 
[(1 ± 6)tt, 7t] and [it, (1 ± S)tt] (square lattice notations, 
unit lattice constant). Even more remarkable is that for 
very low dopings the incommensurability S varies almost 
linearly with the concentration x, but saturates at higher 
dopings. Moreover, the incommensurate peaks broaden 
and weaken in amplitude as the energy increases. This 
incommensurate antiferromagnetism of doped cuprates 
results from special microscopic conditions 3,4 : (1) Cu 
ions situated in a square-planar arrangement and bridged 
by oxygen ions, (2) weak coupling between neighboring 
layers, and (3) doping in such a way that the Fermi 
level lies near the middle of the Cu-0 a* bond. One 
common feature of these doped cuprates is the square- 
planar Cu arrangement. However, some materials with a 
two-dimensional (2D) spin arrangements on non-square 
lattices have been synthetized. In particular, it has 
been reported 5 that there is a class of doped cuprates, 
RCUO2+5, R being a rare-earth element, where the Cu 
ions sites sit not on a square-planar, but on a triangular- 
planar lattice, therefore allowing a test of the geometry 
effect on the spin fluctuations, while retaining some other 
unique microscopic features of the Cu-0 bond. In other 
words, the question is whether the incommensurate mag- 
netic fluctuations observed on the doped square antifer- 
romagnet exist also in the doped triangular antiferromag- 
net? In addition, the doped triangular antiferromagnet, 
where geometric frustration is present, is also of theo- 



retical interest by himself, with many unanswered fasci- 
nating theoretical questions 6 . Historically the undoped 
triangular antiferromagnet was firstly proposed to be a 
model for microscopic realization of the resonating va- 
lence bond spin liquid due to the strong frustration 7 . It 
has been argued 8 that this spin liquid state plays a crucial 
role in the superconductivity of doped cuprates. More- 
over, it has been shown 9 that the doped and undoped 
triangular antiferromagnets present a pairing instability 
in an unconventional channel. 

The incommensurate antiferromagnetism of the doped 
square antiferromagnet has been extensively studied, and 
some fundamentally different microscopic mechanisms 
have been proposed 10-12 to explain the origin of the in- 
commensurate antiferromagnetism, where there is a gen- 
eral consensus that incommensurate antiferromagnetism 
emerges due to doped charge carriers. Recently, it has 
been shown very clearly 13 that if the strong electron cor- 
relation is treated properly and strong spinon-holon in- 
teraction is taken into account, the t-J model can cor- 
rectly reproduce all main features of the incommensu- 
rate antiferromagnetism in the underdoped square anti- 
ferromagnets, including the doping dependence of the in- 
commensurate peak position and the energy dependence 
of the amplitude of these peaks. Since the strong elec- 
tron correlations are present in both doped square and 
triangular antiferromagnets, it is expected that the un- 
conventional incommensurate antiferromagnetism exist- 
ing in the doped square antiferromagnet may also be seen 
in the doped triangular antiferromagnet. In this paper, 
the purpose is to apply the successful approach 13 for the 
doped square lattice antiferromagnet to triangular lat- 
tice. We hope that the information from the present 
work may induce further experimental works in doped 
antiferromagnets on the non-square lattice. 

As in the doped square antiferromagnet, the essential 
physics of the doped triangular antiferromagnet is well 
described by the t-J model on the triangular lattice, 
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with the electron single occupancy local constraint 
J2 a ClaCia < 1, where the summation is over all sites i, 
and for each i, over its nearest-neighbor i), Si = CjaCi/2 
are spin operators with S = (a Xl <7 yi a z ) as Pauli matri- 
ces, and n is the chemical potential. In the t-J model (1), 
the strong electron correlation manifests itself by the lo- 
cal constraint 8 , therefore the crucial requirement is to im- 
pose this local constraint. It has been shown 14 that this 
constraint can be treated properly in analytical calcula- 
tions within the fcrmion-spin theory based on the charge- 
spin separation, where the constrained electron operators 
are decomposed — h\S~ and — h\Sf , where the 
spinless fermion operator hi describes the charge (holon) 
degrees of freedom, while the pseudospin operator Si de- 
scribes the spin (spinon) degrees of freedom, then the low 
energy behavior of the t-J model (1) on the triangular 
lattice in the fermion-spin representation can be written 
as 14 , 

H = -tj^ hih^iS+ST^ + Sr S + + .) + M ]T h \hi 

if} i 

+ Jeff 2_, &i ■ Si+fj, (2) 
if i 

where J c g = [(1 — S) 2 — <f> 2 \, 5 is the hole doping concentra- 
tion, and 4> — (ft-j/ii-)-,-,) is the holon particle-hole order 
parameter. At half-filling, the t-J model is simply the 
Heisenberg model. Many authors have shown that as in 
the square lattice, there is indeed AFLRO in the ground 
state of the AF triangular Heisenberg model 15 , but this 
AFLRO is destroyed more rapidly with increasing dop- 
ings than on the square lattice due to the strong geo- 
metric frustration, then away from the half-filling, there 
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FIG. 1. The Brillouin zone for the triangular system, where 
the antiferromagnetic wave vector Q = [1, v3]. 



is no AFLRO for the doped triangular antiferromagnet, 
i.e., (S*) — 0, and holons move self-consistently in the 
background of the spinon liquid state. 

Within the framework of the fermion-spin theory, 
the spin fluctuations couple only to spinons, but the 
strong correlation between holons and spinon is included 
through the holon's order parameter <fi entering the 
spinon propagator, therefore both spinons and holons are 
responsible for the spin fluctuation. In this case, the in- 
commensurate spin dynamics in the doped square antifer- 
romagnet and integrated spin response in the doped tri- 
angular antiferromagnet have been discussed 13,16 by con- 
sidering the spin fluctuation around the mean-field (MF) 
solution, where the spinon part is treated by a loop ex- 
pansion to second-order. Following their discussions 13,16 , 
we can obtain the dynamical spin structure factor for the 
doped triangular antiferromagnet as, 

S(k,w) = -2[l + n B {uj)]lmD{k,oj) = -2[l + n B (w)] 

X [uj 2 ~uj 2 - 5 fe Re£ s (k,u;)] 2 + [£ fc Im£ s (k, to)} 2 ' (3) 

where the full spinon Green's function, Z? _1 (k, u>) = 
DW-^k,^) - S s (k,w), and the MF spinon Green's 
function D^-'fk.w) = (uj 2 -uj 2 )/B k , ImE s (k,w) and 
ReS s (k, uj) being the imaginary and real parts of the sec- 
ond order spinon self-energy respectively. They are ob- 
tained from the holon bubble, 




FIG. 2. The dynamical spin structure factor in the (k x , k y ) 
plane at doping (a) x = 0.015 and (b) x — 0.06 at temperature 
T = 0.15J and energy uj — 0.05J for parameter t/J = 2.5. 



2 



Fi(k lP ,p') 
u + Cp+p' — £p + ^fe+p' 

w + £p+p' — £p — ^k+p' 
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where B fe = A[(2e Xz + xhfc - (ex + 2**)], A = 
2ZJ eff , e = 1 + 2t(j)/J e ff, 7fe = [cos^ + 
2 cos (k x /2) cos (v / 3A; y /2)]/3, Z is the number of the near- 
est neighbor sites, Fi(k,p,p') = n F {^ p+p ,)[l - n F (t; p )] + 
[1 + n B (uk+p')][nF(k P ) ~ n F {^p+p')], F 2 (k,p,p') = 
n F iU+p')\ l ~ n F{U)\ ~ n B(u k +p>)[n F (£p) - n F (€ p+pl )], 
n F (£,k) and ris(wt) are the fermion and boson distri- 
bution functions, respectively, the MF holon excitation 
£fc = 2xtZ-fk + M> and MF spinon excitation, w\ = 
A 2 (Ai 7 2 + A 2 -fk + A 3 ) with A-i = ae( X /2 + e Xz ), A 2 = 
e[(l - Z)a(e X /2 + Xx )/Z - a(C z + C/2) - (1 - a)/{2Z% 
A 3 = a(C z + e 2 C/2) + (1 - a){\ + e 2 )/(AZ) - ae( X /2 + 



i+r/i 

and C' z = 



eXz)/Z, the spinon correlation functions x 

Xz = (SfSi +n ), c = (i/z 2 )£-,<s+.s.- +j? 

(1/Z 2 ) ^fj', (S" + ^Sf + ~,). In order to satisfy the sum ride 

for the correlation function (S^S//) = 1/2 in the ab- 
sence of AFLRO, a decoupling parameter a has been 
introduced in the self-consistent MF calculation, which 
can be regarded as the vertex correction 17 . 

The Brillouin zone of the triangular system is shown 
in Fig. 1, where the AF wave vector Q = [1, \/3] (here- 
after we use the units of [27r/3, 27r/3]). In Fig. 2, we plot 
the dynamical spin structure factor S(k, u) in the [k x , k y ] 
plane at doping (a) x = 0.015 and (b) x — 0.06 with tem- 
perature T = 0.15J and energy u> = 0.05 J for parameter 
t/J = 2.5. From Fig. 2, we find that a commensurate- 
incommensurate transition in the spin fluctuation Q vec- 
tor occurs with doping, i.e., the commensurate peak 
near half-filling (x < 0.015) is split into six peaks in 
the underdoped regime, while the positions of these split 
peaks are incommensurate with the underlying lattice, 
and correspond to six 2D rods at [(1 — S x ), (y/3 ± 8 y )], 
[(1 - S' X ),(V3 ± 6' y )], and [(1 - 8"), (Vs ± 5^)] with 
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FIG. 3. The doping dependence of the incommensurability 
S(x) of the antiferromagnetic fluctuations. 
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low temperatures, the incommensurate peaks are very 
sharp at lower energies, which means that these excita- 
tions have a dynamical coherence length that is larger 
than the instantaneous correlation length. The present 
dynamical spin structure factor spectrum S(k,u>) has 
been used to extract the doping dependence of the incom- 
mensurability parameter 8(x), defined as the deviation of 
the peak position from the AF wave vector Q, and the 
result is plotted in Fig. 3. Our result shows that 6(x) 
increases with the hole concentration in lower dopings, 
but it saturates at higher dopings, which is qualitatively 
similar to the results for the square antiferromagnet 3 ' 4,13 . 

In order to study the effects of energy and tempera- 
ture on the incommensurate spin fluctuation, we have 
calculated the dynamical spin structure factor S(k,uj) 
at different temperatures and energies, and the result at 
doping x = 0.06 for parameter t/J = 2.5, temperature 
T = 0.15J and energy ui = 0.1J is shown in Fig. 4. 
In comparison with Fig. 2(b) for the same set of pa- 
rameters except for lu = 0.05J, we see that although 
the positions of the incommensurate peaks are almost 
energy independent, the weight of these peaks decreases 
with increasing energy, and tends to vanish at high en- 
ergies. This reflects that the inverse lifetime of the spin 




FIG. 4. The dynamical spin structure factor in the (k x , k y ) 
plane at doping x = 0.06 for parameter t/J = 2.5 and energy 
u) — 0.1 J at temperature T = 0.15J. 




FIG. 5. The dynamical spin structure factor in the (k x , k y ) 
plane at doping x = 0.06 for parameter t/J = 2.5 and energy 
ui — 0.05 J at temperature T = 0.3J. 
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excitations increases with increasing energy. In corre- 
spondence, the dynamical spin structure factor S(k, ui) 
at doping x — 0.06 for parameter t/J — 2.5 in energy 
ui = 0.05 J and temperature T = 0.3J is plotted in Fig. 
5. Comparing with Fig. 2(b) for the same set of parame- 
ters except for T = 0.15 J, we find that the peak weight is 
suppressed with increasing temperature. Our results also 
indicate that although the positions of the incommensu- 
rate peaks in the doped triangular antifcrromagnets are 
much different from these in the doped square antiferro- 
magnets due to the geometric frustration, the behavior of 
the energy and temperature dependence of the magnetic 
fluctuation is qualitatively similar to these in the doped 
square antifcrromagnets 3 ' 4,13 . 

As in the case of the doped square antiferromagnet 13 , 
the effect of holons on the spinons due to the strong 
spinon-holon interaction is critical in determining the 
characteristic feature of the incommensurate spin fluc- 
tuation in the doped triangular antiferromagnet. This is 
reflected by the renormalization of the spinon excitation 
E 2 = lj 2 + B k ReY, s (k,E k ) in Eq. (3). Figure 6 shows 
this renormalized spinon excitation for doping x = 0.015 
(solid line), x = 0.03 (dashed line), and x = 0.06 (dot- 
dashed line), and for t/J = 2.5 and T = 0.15 J. An over 
damping around the AF wave vector Q occurs for doping 
x > 0.015. The incommensurate peaks emerge in the dy- 




FIG. 6. The spinon excitation spectrum at doping 
x = 0.015 (solid line), x = 0.03 (dashed line), and x = 0.06 
(dot-dashed line) in parameter t/J — 2.5 with temperature 
T = 0.15J. 



namical spin structure factor S*(k, u>) when the incoming 
neutron energy u> is equal to the renormalized spin excita- 
tion E k , i.e. W(k s ,u/) = [lu 2 -Lul s -B k5 ReY, s (ks,u})] 2 = 
(lj 2 — Ef, s ) 2 ~ for some critical wave vectors k,5 (posi- 
tions of the incommensurate peaks). Then the weight of 
these peaks is determined by the imaginary part of the 
spinon self-energy 1/Im5] s (k,5, u>). Therefore near half- 
filling, the spin excitations are centered around the AF 
wave vector Q, so the commensurate AF peak appears 
there. Upon doping, the holes disturb the AF back- 
ground self-consistently, and induce over damping of the 
spinon excitation around the AF wave vector Q, lead- 
ing to incommensurate antifcrromagnetism. Since the 
weight of the incommensurate peaks is determined by 
the imaginary part of the spinon self-energy, it is then 
understandable that they are suppressed as energy and 
temperature are increased. Moreover, the symmetry of 
the incommensurate pattern is determined by the lattice 
symmetry in the spinon self-energy renormalization due 
to holons. We have plotted the function W(k,oj) along 
the line shown in the inset of Fig. 7 at doping x = 0.06 
for parameter t/J = 2.5 and temperature T = 0.15 J 
and energy lo — 0.05 J in Fig. 7. There is a strong an- 
gular dependence with minima at [(1 — S x ), (\/3 — S y )], 
[(l — 5' x ), (\/3— S' y )]. These are exactly the positions of the 
incommensurate peaks determined by the dispersion of 
very well defined renormalized spin excitations. This in- 
commensurate pattern is much different from that in the 
doped square antiferromagnet 13 , since the lattice symme- 
try in the triangular lattice is much different from that 
in the square lattice. 

In summary, we have discussed the incommensurate 
spin fluctuation of the doped triangular antiferromagnet 
within the t-J model based on the fermion-spin theory. It 
is shown that the commensurate peak near the half-filling 
is split into six incommensurate peaks in the underdoped 
and optimally doped regimes. The incommensurability 
increases with the hole concentration at lower dopings, 
and saturates at higher dopings. Although the incom- 
mensurability is almost energy independent, the weight 
of these incommensurate peaks decreases with energy and 
temperature. 
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FIG. 7. Function W(k,w) at doping x = 0.06 for parameter 
t/J = 2.5 and energy to — 0.05J at temperatures T = 0.15J. 
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